Detecting Compliance with Business Rules in Ontology-Based Process Modeling

Extending business processes with semantic annotations has gained recent attention. This comprises relating process elements to ontology elements in order to create a shared conceptual and terminological understanding. In business process modeling, processes may have to adhere to a multitude of rules. A common way to detect compliance automatedly is studying the artifact of the process model itself. However, if an ontology exists as an additional artifact, it may prove beneficial to exploit this structure for compliance detection, as it provides a rich specification of the business process. We therefore propose an approach that models a rules-layer ontop of an ontology. Said rules-layer is implemented by a logic program and can be used to reason about the compliance of an underlying ontology. Our approach allows ad-hoc access to external ontologies, other than similar approaches that are reliant on a redundant logical representation of process model elements

Resolving Inconsistencies in Declarative Process Models based on Culpability Measurement

Contrary to traditional process models, declarative process models define a set of declarative constraints to specify the behavior which a process should adhere to. In the scope of process mining, declarative process discovery aims to derive such constraint sets from event logs. Here, a problem for current discovery techniques is that of inconsistency. That is, dependent of certain event log characteristics, the derived constraint set may contain contradictory constraints. This in turn however makes the discovered model unusable, as contradictory constraints make it impossible to execute declarative process models, thus hampering previous process discovery efforts. In this work, we present an approach for resolving inconsistencies in declarative process models, based on methods from the scientific field of inconsistency measurement. We introduce our approach algorithm and evaluate its feasibility with data sets of the BPI Challenge 2017

Towards Intelligent Chatbots for Customer Care - Practice-Based Requirements for a Research Agenda

Chatbots bare a great potential to save efforts and costs in customer care through service automation. Current results are however still at an early stage in functionality and not widely attainable. Here, developing a new form of intelligent chatbots is a current challenge still under review. While there have been numerous proposals for future work, virtually all agenda-setting contributions are solely based on scientific literature. This is unsatisfactory from both an academic and practical perspective, as the industrial view on the future of chatbots seems to be neglected. Therefore, this work explores how professional experts see the future of intelligent chatbots for customer care and suggests how practice can guide research based on an expert panel with 17 industrial partners. Our work identifies research opportunities based on the demands and views of key practitioners by pin-pointing expected trends. Furthermore, based on the expert opinions, we derive guidelines for organizations which state key factors that should be considered in the development or adoption of chatbots in customer care

Effects of Visualization Techniques on Understanding Inconsistencies in Automated Decision-Making

The automation of business processes and decision-making have received major interest from practice and academia. As automation allows to execute more processes (cases), monitoring automated decision-making is currently evolving into a big data analytics problem for companies. Thus, not only monitoring insights themselves, but also an effective use of such insights become important. In this context, the speed and ability to interpret data is closely related to the visualization of metrics and data. While various approaches for quantitative insights on automated decision-making have been proposed, there is currently no evidence as to how the specific visualization of such metrics helps companies to create more value from their data. In this report, we therefore present the results of an empirical experiment analyzing the cognitive effects of different visualization techniques for quantitative insights on understanding inconsistencies in automated decision-making data

Effects of Quantitative Measures on Understanding Inconsistencies in Business Rules

Business Rules have matured to an important aspect in the development of organizations, encoding company knowledge as declarative constraints, aimed to ensure compliant business. The management of business rules is widely acknowledged as a challenging task. A problem here is a potential inconsistency ofbusiness rules, as business rules are often created collaboratively. To support companies in managing inconsistency, many works have suggested that a quantification of inconsistencies could provide valuable insights. However, the actual effects of quantitative insights in business rules management have not yet been evaluated. In this work, we present the results of an empirical experiment using eye-tracking and other performance measures to analyze the effects of quantitative measures on understanding inconsistencies in business rules. Our results indicate that quantitative measures are associated with better understanding accuracy, understanding efficiency and less mental effort in business rules management

Advanced Auditing of Run-Time Conflicts in Declarative Process Models using Time Series Clustering

We present a novel approach for auditing conflicts between declarative constraints that arise during process execution, i.e., relative to observed traces. As a main advantage, taking a post-execution perspective allows to consider all observed traces and their interrelations, and to assess conflicts from a global perspective. Our approach allows to classify and prioritize conflicts as a basis for re-modelling, e.g., which conflicts are an outlier, and which require an urgent change to the model. Also, our approach provides means for quantitative root-cause analysis, i.e., prioritizing which rules need to be changed. We implement our approach and show that it can be applied in settings of industrial scale by means of runtime experiments with real-life data-sets

Acute toxicity, brine shrimp cytotoxicity and relaxant activity of fruits of callistemon citrinus curtis

<p>Abstract</p> <p>Background</p> <p><it>Callistemon citrinus </it>Curtis belongs to family Myrtaceae that has a great medicinal importance. In our previous work, fruits of <it>Callistemon citrinus </it>were reported to have relaxant (antispasmodic) activity. The current work describes the screening of fractions of the crude methanol extract for tracing spasmolytic constituents so that it shall help us for isolation of bioactive compounds. Acute toxicity and brine shrimp cytotoxicity of crude methanol extract are also performed to standardize it.</p> <p>Methods</p> <p>The crude methanol extract was obtained by maceration with distilled water (500 ml) three times and fractionated successively with <it>n-</it>hexane, chloroform, ethyl acetate and <it>n-</it>butanol (300 ml of each solvent). Phytochemical analysis for crude methanol extract was performed. Acute toxicity studies were performed in mice. Brine shrimp cytotoxicity studies were performed to determine its cytotoxicity and standardize it. In other series of experiments, rabbits' jejunum preparations were used in screening for possible relaxant activities of various fractions. They were applied in concentrations of 0.01, 0.03, 0.1, 0.3, 1.0, 3.0, 5.0 and 10.0 mg/ml on spontaneous rabbits' jejunum preparations. In similar fashion, fractions were also tested on KCl (80 mM) -induced contractions. Calcium chloride curves were constructed in K-rich Tyrode's solution. The effects of various fractions were tested on calcium chloride curves at concentrations 1.0, 3.0, 5.0 and 10.0 mg/ml. Curves of verapamil used as reference drug at concentration 0.1 μM and 0.3 μM were also constructed. The curves were compared with their respective controls for possible right shift.</p> <p>Results</p> <p>Methanol extract tested strongly positive for saponins and tannins. However, it tested mild positive for presence of proteins, amino acids, carbohydrates and phenolic compounds. LD₅₀value for crude methanol extract is 476.25 ± 10.3 (470-481, n = 4) mg/ml. Similarly, EC₅₀value for brine shrimp cytotoxicity is 65.5 ± 7.28 (60.8- 69.4, n = 4) mg/ml. All the fractions relaxed the spontaneous and KCl-induced contractions. EC₅₀values (mg/ml) for effects of ethyl acetate fraction on spontaneous and KCl induced contractions are 2.62 ± 0.78 (2.15-3.0, n = 4) and 3.72 ± 0.86 (3.38-4.28, n = 4) respectively. Respective EC₅₀values (mg/ml) for <it>n-</it>butanol fraction are 3.59 ± 0.2(3.07-3.9, n = 4) for spontaneous, and 5.57 ± 0.2 (5.07-6.11, n = 4) for KCl- induced contractions. EC₅₀value for control calcium chloride curve (without extract) is -2.73 ± 0.19 (-2.6 - -2.81, n = 4) while EC₅₀for curves treated with 5.0 mg/ml of chloroform is -2.22 ± 0.02 (-2.16 - -2.3, n = 4). EC₅₀value for ethyl acetate treated (1.0 mg/ml) tissues is -1.95 ± 0.10 (-1.88 - -2.0, n = 4) <it>vs</it>. control EC₅₀= -2.71 ± 0.08 (-2.66 - -2.76, n = 4). All the fractions, except <it>n-</it>hexane, showed a right shift like that of verapamil (EC₅₀= -1.72 ± 0.15 (-1.62 - -1.8, n = 4) vs. Control EC₅₀= -2.41 ± 0.06 (-2.38 - - 2.44, n = 4), a standard drug that blocks voltage operated calcium channels.</p> <p>Conclusion</p> <p>Relaxant constituents were more concentrated in ethylacetate fraction followed by chloroform, <it>n -</it>butanol and aqueous fractions that warrant for its isolation. The crude methanol extract is safe at concentration 250 mg/ml or below and results of brine shrimp cytotoxicity assay imply the plant specie may be a source of cytotoxic agents.</p

Human Practice. Digital Ecologies. Our Future. : 14. Internationale Tagung Wirtschaftsinformatik (WI 2019) : Tagungsband

Erschienen bei: universi - Universitätsverlag Siegen. - ISBN: 978-3-96182-063-4Aus dem Inhalt: Track 1: Produktion & Cyber-Physische Systeme Requirements and a Meta Model for Exchanging Additive Manufacturing Capacities Service Systems, Smart Service Systems and Cyber- Physical Systems—What’s the difference? Towards a Unified Terminology Developing an Industrial IoT Platform – Trade-off between Horizontal and Vertical Approaches Machine Learning und Complex Event Processing: Effiziente Echtzeitauswertung am Beispiel Smart Factory Sensor retrofit for a coffee machine as condition monitoring and predictive maintenance use case Stakeholder-Analyse zum Einsatz IIoT-basierter Frischeinformationen in der Lebensmittelindustrie Towards a Framework for Predictive Maintenance Strategies in Mechanical Engineering - A Method-Oriented Literature Analysis Development of a matching platform for the requirement-oriented selection of cyber physical systems for SMEs Track 2: Logistic Analytics An Empirical Study of Customers’ Behavioral Intention to Use Ridepooling Services – An Extension of the Technology Acceptance Model Modeling Delay Propagation and Transmission in Railway Networks What is the impact of company specific adjustments on the acceptance and diffusion of logistic standards? Robust Route Planning in Intermodal Urban Traffic Track 3: Unternehmensmodellierung & Informationssystemgestaltung (Enterprise Modelling & Information Systems Design) Work System Modeling Method with Different Levels of Specificity and Rigor for Different Stakeholder Purposes Resolving Inconsistencies in Declarative Process Models based on Culpability Measurement Strategic Analysis in the Realm of Enterprise Modeling – On the Example of Blockchain-Based Initiatives for the Electricity Sector Zwischenbetriebliche Integration in der Möbelbranche: Konfigurationen und Einflussfaktoren Novices’ Quality Perceptions and the Acceptance of Process Modeling Grammars Entwicklung einer Definition für Social Business Objects (SBO) zur Modellierung von Unternehmensinformationen Designing a Reference Model for Digital Product Configurators Terminology for Evolving Design Artifacts Business Role-Object Specification: A Language for Behavior-aware Structural Modeling of Business Objects Generating Smart Glasses-based Information Systems with BPMN4SGA: A BPMN Extension for Smart Glasses Applications Using Blockchain in Peer-to-Peer Carsharing to Build Trust in the Sharing Economy Testing in Big Data: An Architecture Pattern for a Development Environment for Innovative, Integrated and Robust Applications Track 4: Lern- und Wissensmanagement (e-Learning and Knowledge Management) eGovernment Competences revisited – A Literature Review on necessary Competences in a Digitalized Public Sector Say Hello to Your New Automated Tutor – A Structured Literature Review on Pedagogical Conversational Agents Teaching the Digital Transformation of Business Processes: Design of a Simulation Game for Information Systems Education Conceptualizing Immersion for Individual Learning in Virtual Reality Designing a Flipped Classroom Course – a Process Model The Influence of Risk-Taking on Knowledge Exchange and Combination Gamified Feedback durch Avatare im Mobile Learning Alexa, Can You Help Me Solve That Problem? - Understanding the Value of Smart Personal Assistants as Tutors for Complex Problem Tasks Track 5: Data Science & Business Analytics Matching with Bundle Preferences: Tradeoff between Fairness and Truthfulness Applied image recognition: guidelines for using deep learning models in practice Yield Prognosis for the Agrarian Management of Vineyards using Deep Learning for Object Counting Reading Between the Lines of Qualitative Data – How to Detect Hidden Structure Based on Codes Online Auctions with Dual-Threshold Algorithms: An Experimental Study and Practical Evaluation Design Features of Non-Financial Reward Programs for Online Reviews: Evaluation based on Google Maps Data Topic Embeddings – A New Approach to Classify Very Short Documents Based on Predefined Topics Leveraging Unstructured Image Data for Product Quality Improvement Decision Support for Real Estate Investors: Improving Real Estate Valuation with 3D City Models and Points of Interest Knowledge Discovery from CVs: A Topic Modeling Procedure Online Product Descriptions – Boost for your Sales? Entscheidungsunterstützung durch historienbasierte Dienstreihenfolgeplanung mit Pattern A Semi-Automated Approach for Generating Online Review Templates Machine Learning goes Measure Management: Leveraging Anomaly Detection and Parts Search to Improve Product-Cost Optimization Bedeutung von Predictive Analytics für den theoretischen Erkenntnisgewinn in der IS-Forschung Track 6: Digitale Transformation und Dienstleistungen Heuristic Theorizing in Software Development: Deriving Design Principles for Smart Glasses-based Systems Mirroring E-service for Brick and Mortar Retail: An Assessment and Survey Taxonomy of Digital Platforms: A Platform Architecture Perspective Value of Star Players in the Digital Age Local Shopping Platforms – Harnessing Locational Advantages for the Digital Transformation of Local Retail Outlets: A Content Analysis A Socio-Technical Approach to Manage Analytics-as-a-Service – Results of an Action Design Research Project Characterizing Approaches to Digital Transformation: Development of a Taxonomy of Digital Units Expectations vs. Reality – Benefits of Smart Services in the Field of Tension between Industry and Science Innovation Networks and Digital Innovation: How Organizations Use Innovation Networks in a Digitized Environment Characterising Social Reading Platforms— A Taxonomy-Based Approach to Structure the Field Less Complex than Expected – What Really Drives IT Consulting Value Modularity Canvas – A Framework for Visualizing Potentials of Service Modularity Towards a Conceptualization of Capabilities for Innovating Business Models in the Industrial Internet of Things A Taxonomy of Barriers to Digital Transformation Ambidexterity in Service Innovation Research: A Systematic Literature Review Design and success factors of an online solution for cross-pillar pension information Track 7: IT-Management und -Strategie A Frugal Support Structure for New Software Implementations in SMEs How to Structure a Company-wide Adoption of Big Data Analytics The Changing Roles of Innovation Actors and Organizational Antecedents in the Digital Age Bewertung des Kundennutzens von Chatbots für den Einsatz im Servicedesk Understanding the Benefits of Agile Software Development in Regulated Environments Are Employees Following the Rules? On the Effectiveness of IT Consumerization Policies Agile and Attached: The Impact of Agile Practices on Agile Team Members’ Affective Organisational Commitment The Complexity Trap – Limits of IT Flexibility for Supporting Organizational Agility in Decentralized Organizations Platform Openness: A Systematic Literature Review and Avenues for Future Research Competence, Fashion and the Case of Blockchain The Digital Platform Otto.de: A Case Study of Growth, Complexity, and Generativity Track 8: eHealth & alternde Gesellschaft Security and Privacy of Personal Health Records in Cloud Computing Environments – An Experimental Exploration of the Impact of Storage Solutions and Data Breaches Patientenintegration durch Pfadsysteme Digitalisierung in der Stressprävention – eine qualitative Interviewstudie zu Nutzenpotenzialen User Dynamics in Mental Health Forums – A Sentiment Analysis Perspective Intent and the Use of Wearables in the Workplace – A Model Development Understanding Patient Pathways in the Context of Integrated Health Care Services - Implications from a Scoping Review Understanding the Habitual Use of Wearable Activity Trackers On the Fit in Fitness Apps: Studying the Interaction of Motivational Affordances and Users’ Goal Orientations in Affecting the Benefits Gained Gamification in Health Behavior Change Support Systems - A Synthesis of Unintended Side Effects Investigating the Influence of Information Incongruity on Trust-Relations within Trilateral Healthcare Settings Track 9: Krisen- und Kontinuitätsmanagement Potentiale von IKT beim Ausfall kritischer Infrastrukturen: Erwartungen, Informationsgewinnung und Mediennutzung der Zivilbevölkerung in Deutschland Fake News Perception in Germany: A Representative Study of People’s Attitudes and Approaches to Counteract Disinformation Analyzing the Potential of Graphical Building Information for Fire Emergency Responses: Findings from a Controlled Experiment Track 10: Human-Computer Interaction Towards a Taxonomy of Platforms for Conversational Agent Design Measuring Service Encounter Satisfaction with Customer Service Chatbots using Sentiment Analysis Self-Tracking and Gamification: Analyzing the Interplay of Motivations, Usage and Motivation Fulfillment Erfolgsfaktoren von Augmented-Reality-Applikationen: Analyse von Nutzerrezensionen mit dem Review-Mining-Verfahren Designing Dynamic Decision Support for Electronic Requirements Negotiations Who is Stressed by Using ICTs? A Qualitative Comparison Analysis with the Big Five Personality Traits to Understand Technostress Walking the Middle Path: How Medium Trade-Off Exposure Leads to Higher Consumer Satisfaction in Recommender Agents Theory-Based Affordances of Utilitarian, Hedonic and Dual-Purposed Technologies: A Literature Review Eliciting Customer Preferences for Shopping Companion Apps: A Service Quality Approach The Role of Early User Participation in Discovering Software – A Case Study from the Context of Smart Glasses The Fluidity of the Self-Concept as a Framework to Explain the Motivation to Play Video Games Heart over Heels? An Empirical Analysis of the Relationship between Emotions and Review Helpfulness for Experience and Credence Goods Track 11: Information Security and Information Privacy Unfolding Concerns about Augmented Reality Technologies: A Qualitative Analysis of User Perceptions To (Psychologically) Own Data is to Protect Data: How Psychological Ownership Determines Protective Behavior in a Work and Private Context Understanding Data Protection Regulations from a Data Management Perspective: A Capability-Based Approach to EU-GDPR On the Difficulties of Incentivizing Online Privacy through Transparency: A Qualitative Survey of the German Health Insurance Market What is Your Selfie Worth? A Field Study on Individuals’ Valuation of Personal Data Justification of Mass Surveillance: A Quantitative Study An Exploratory Study of Risk Perception for Data Disclosure to a Network of Firms Track 12: Umweltinformatik und nachhaltiges Wirtschaften Kommunikationsfäden im Nadelöhr – Fachliche Prozessmodellierung der Nachhaltigkeitskommunikation am Kapitalmarkt Potentiale und Herausforderungen der Materialflusskostenrechnung Computing Incentives for User-Based Relocation in Carsharing Sustainability’s Coming Home: Preliminary Design Principles for the Sustainable Smart District Substitution of hazardous chemical substances using Deep Learning and t-SNE A Hierarchy of DSMLs in Support of Product Life-Cycle Assessment A Survey of Smart Energy Services for Private Households Door-to-Door Mobility Integrators as Keystone Organizations of Smart Ecosystems: Resources and Value Co-Creation – A Literature Review Ein Entscheidungsunterstützungssystem zur ökonomischen Bewertung von Mieterstrom auf Basis der Clusteranalyse Discovering Blockchain for Sustainable Product-Service Systems to enhance the Circular Economy Digitale Rückverfolgbarkeit von Lebensmitteln: Eine verbraucherinformatische Studie Umweltbewusstsein durch audiovisuelles Content Marketing? Eine experimentelle Untersuchung zur Konsumentenbewertung nachhaltiger Smartphones Towards Predictive Energy Management in Information Systems: A Research Proposal A Web Browser-Based Application for Processing and Analyzing Material Flow Models using the MFCA Methodology Track 13: Digital Work - Social, mobile, smart On Conversational Agents in Information Systems Research: Analyzing the Past to Guide Future Work The Potential of Augmented Reality for Improving Occupational First Aid Prevent a Vicious Circle! The Role of Organizational IT-Capability in Attracting IT-affine Applicants Good, Bad, or Both? Conceptualization and Measurement of Ambivalent User Attitudes Towards AI A Case Study on Cross-Hierarchical Communication in Digital Work Environments ‘Show Me Your People Skills’ - Employing CEO Branding for Corporate Reputation Management in Social Media A Multiorganisational Study of the Drivers and Barriers of Enterprise Collaboration Systems-Enabled Change The More the Merrier? The Effect of Size of Core Team Subgroups on Success of Open Source Projects The Impact of Anthropomorphic and Functional Chatbot Design Features in Enterprise Collaboration Systems on User Acceptance Digital Feedback for Digital Work? Affordances and Constraints of a Feedback App at InsurCorp The Effect of Marker-less Augmented Reality on Task and Learning Performance Antecedents for Cyberloafing – A Literature Review Internal Crowd Work as a Source of Empowerment - An Empirical Analysis of the Perception of Employees in a Crowdtesting Project Track 14: Geschäftsmodelle und digitales Unternehmertum Dividing the ICO Jungle: Extracting and Evaluating Design Archetypes Capturing Value from Data: Exploring Factors Influencing Revenue Model Design for Data-Driven Services Understanding the Role of Data for Innovating Business Models: A System Dynamics Perspective Business Model Innovation and Stakeholder: Exploring Mechanisms and Outcomes of Value Creation and Destruction Business Models for Internet of Things Platforms: Empirical Development of a Taxonomy and Archetypes Revitalizing established Industrial Companies: State of the Art and Success Principles of Digital Corporate Incubators When 1+1 is Greater than 2: Concurrence of Additional Digital and Established Business Models within Companies Special Track 1: Student Track Investigating Personalized Price Discrimination of Textile-, Electronics- and General Stores in German Online Retail From Facets to a Universal Definition – An Analysis of IoT Usage in Retail Is the Technostress Creators Inventory Still an Up-To-Date Measurement Instrument? Results of a Large-Scale Interview Study Application of Media Synchronicity Theory to Creative Tasks in Virtual Teams Using the Example of Design Thinking TrustyTweet: An Indicator-based Browser-Plugin to Assist Users in Dealing with Fake News on Twitter Application of Process Mining Techniques to Support Maintenance-Related Objectives How Voice Can Change Customer Satisfaction: A Comparative Analysis between E-Commerce and Voice Commerce Business Process Compliance and Blockchain: How Does the Ethereum Blockchain Address Challenges of Business Process Compliance? Improving Business Model Configuration through a Question-based Approach The Influence of Situational Factors and Gamification on Intrinsic Motivation and Learning Evaluation von ITSM-Tools für Integration und Management von Cloud-Diensten am Beispiel von ServiceNow How Software Promotes the Integration of Sustainability in Business Process Management Criteria Catalog for Industrial IoT Platforms from the Perspective of the Machine Tool Industry Special Track 3: Demos & Prototyping Privacy-friendly User Location Tracking with Smart Devices: The BeaT Prototype Application-oriented robotics in nursing homes Augmented Reality for Set-up Processe Mixed Reality for supporting Remote-Meetings Gamification zur Motivationssteigerung von Werkern bei der Betriebsdatenerfassung Automatically Extracting and Analyzing Customer Needs from Twitter: A “Needmining” Prototype GaNEsHA: Opportunities for Sustainable Transportation in Smart Cities TUCANA: A platform for using local processing power of edge devices for building data-driven services Demonstrator zur Beschreibung und Visualisierung einer kritischen Infrastruktur Entwicklung einer alltagsnahen persuasiven App zur Bewegungsmotivation für ältere Nutzerinnen und Nutzer A browser-based modeling tool for studying the learning of conceptual modeling based on a multi-modal data collection approach Exergames & Dementia: An interactive System for People with Dementia and their Care-Network Workshops Workshop Ethics and Morality in Business Informatics (Workshop Ethik und Moral in der Wirtschaftsinformatik – EMoWI’19) Model-Based Compliance in Information Systems - Foundations, Case Description and Data Set of the MobIS-Challenge for Students and Doctoral Candidates Report of the Workshop on Concepts and Methods of Identifying Digital Potentials in Information Management Control of Systemic Risks in Global Networks - A Grand Challenge to Information Systems Research Die Mitarbeiter von morgen - Kompetenzen künftiger Mitarbeiter im Bereich Business Analytics Digitaler Konsum: Herausforderungen und Chancen der Verbraucherinformati

Large expert-curated database for benchmarking document similarity detection in biomedical literature search

Document recommendation systems for locating relevant literature have mostly relied on methods developed a decade ago. This is largely due to the lack of a large offline gold-standard benchmark of relevant documents that cover a variety of research fields such that newly developed literature search techniques can be compared, improved and translated into practice. To overcome this bottleneck, we have established the RElevant LIterature SearcH consortium consisting of more than 1500 scientists from 84 countries, who have collectively annotated the relevance of over 180 000 PubMed-listed articles with regard to their respective seed (input) article/s. The majority of annotations were contributed by highly experienced, original authors of the seed articles. The collected data cover 76% of all unique PubMed Medical Subject Headings descriptors. No systematic biases were observed across different experience levels, research fields or time spent on annotations. More importantly, annotations of the same document pairs contributed by different scientists were highly concordant. We further show that the three representative baseline methods used to generate recommended articles for evaluation (Okapi Best Matching 25, Term Frequency-Inverse Document Frequency and PubMed Related Articles) had similar overall performances. Additionally, we found that these methods each tend to produce distinct collections of recommended articles, suggesting that a hybrid method may be required to completely capture all relevant articles. The established database server located at https://relishdb.ict.griffith.edu.au is freely available for the downloading of annotation data and the blind testing of new methods. We expect that this benchmark will be useful for stimulating the development of new powerful techniques for title and title/abstract-based search engines for relevant articles in biomedical research.Peer reviewe